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ABSTRACT 

The National Center on Educational Outcomes (NCEO) 
was established to work with states and other policy groups to 
develop a model of educational outcomes for students with 
disabilities and to generate a list of indicators of these outcomes* 
NCEO developed a technique named Multi-Attribute Consensus Building 
(MACB) to build consensus among educators, policymakers, 
administrators, parents, advocates, and other stakeholders* MACB is a 
quantitative, objective approach for determining a small group's 
opinion about the importance of each item in a list, through three 
stages: generation of input, consensus working session, and synthesis 
of consensus* NCEO produced large lists of outcomes and indicators 
and used MACB to determine how important these indicators were to 
various groups* NCEO used the ratings to determine which indicators 
and outcomes to use in the model at the school completion, 
postschool, and early childhood levels* Following a description of 
the MACB approach, this report discusses equipment needed, technical 
and logistical issues, ways to modify the MACB process, and 
advantages and limitations* Figures and appendixes present: (1) a 
conceptual model of outcomes; (2) MACB stages and tasks to 
accomplish; (3) a table showing decisions made during the process of 
synthesizing consensu:; (4) a handout on the consensus-building 
process; (5) an example of an indicator's importance ratings 
spreadsheet ; and (6) equipment configurations ♦ (Contains 1 1 
references *) (JDD) 
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Consensus Building: 
A Process for Selecting Educational 
Outcomes and Indicators 



by Michael Vandenvood, James Ysseldyke and Martha Thurlozo 



In the decade since A Nation at 
Risk (National Commission on 
Excellence in Education, 1983), 
there has been a great deal of 
effort put into developing an 
evaluation system to provide 
educators, policymakers and 
the general public with infor- 
mation on the condition of pub- 
lic education. The search for 
this system has shifted from 
examining the process of edu- 
cation to identifying the desired 
educational outcomes of 

i schooling and indicators of 

^ these outcomes. 

<^ By using outcomes and indica- 

* tors, decisions about the 
^ educational system can be data- 
J based, with the goal of imp.ov- 
ing instruction and student and 
system outcomes. Although 



there is an increasing national 
focus on outcomes and 
indicators, very little work has 
been done to identify outcomes 
for students with disabilities. 

Because of this, the National 
Center on Educational 
Outcomes (NCEO) was estab- 
lished in October 1990, to work 
with states and other policy 
groups to develop a model of 
educational outcomes for stu- 
dents with disabilities and to 
generate a list of indicators of 
these outcomes. 

In order to develop a useful 
and meaningful system for 
assessing the condition of edu- 
cation in this countrv, NCEO 
and others believed consensus 
needed to be created among 



various groups of stakeholders 
about the specific outcomes and 
indicators used in the model 
(Blank, 1993; Ysseldyke, 
Thurlow, Bruininks, Gilman, 
Deno, McGrew, & Shriner, 
1992). Therefore, in its effort to 
develop a system of indicators, 
NCEO realized it had to use a 
process that would build 
consensus among educators, 
policymakers, administrators, 
parents, advocates, and other 
stakeholders. 
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Multi-Attribute Consensus Building (MACB) 

MACB is a quantitative, objective approach for determining a small 
group's opinion about the importance of each item in a list. The 
group members begin by submitting items they believe should be 
included in the list. 

Next, each member of the group rates the importance of each item 
in the list on a scale from 0 to 100. To create consensus, the facilita- 
tor presents each member's ratings and averages for each item, and 
then leads discussion around items that have a relatively large 
range or variance in ratings. Computer technology records ratings 
and calculates descriptive statistical information about the ratings. 
It can also be used to display changes to items that may be made 
prior to the rating of those items. 

With this approach, discussion focuses on presenting and under- 
standing various viewpoints regarding specific items about which 
members disagree. After hearing different viewpoints, members 
may change their ratings, giving an objective indication of the 
extent to which the group agrees on the importance of each item. 
The computer technology enables participants to view the effects of 
adjusted ratings immediately upon changing ratings. 



To identify the process it would 
use, NCEO reviewed several 
different approaches and decid- 
ed to modify an evaluation sys- 
tem used in the management 
sciences. 

Multi-Attribute Utility (MAU) 
analysis, is an evaluation model 
designed to facilitate decision 
making about several different 
options or alternatives. (See 
Lewis, Erickson, Johnson, and 
Bruininks, 1991, for detailed 
description of MAU.) 

Using the MAU structure, 
NCEO created a consensus- 
building technique named 
Multi-Attribute Consensus 
Building (MACB). Through this 
process, outcomes and indica- 
tors w r ere developed for the 
NCEO model at the school 
completion (Ysseldyke, 
Thurlow, & Gilman, 1993d) and 
post-school levels (Ysseldyke, 
Thurlow, & Gilman, 1993c), and 
for the early childhood levels at 
age three (Ysseldyke, Thurlow, 
& Gilman, 1993a) and age six 
(Ysseldyke, Thurlow, & 
Gilman, 1993b). 



groups. MACB working ses- 
sions have been held with sev- 
eral groups of stakeholders. 
(For a more detailed descrip- 
tion of the use of MACB at 
NCEO see Ysseldyke and 
Thurlow, 1993a.) After gaining 
input from these groups, NCEO 
used their ratings to determine 
which indicators and outcomes 
to use in the model. NCEO also 
produced a self-study guide to 
help states and school districts 
develop outcomes and indica- 
tors based on its model 
(Ysseldyke & Thurlow, 1993b). 

Description of the Process 

Three general stages are 
involved in using the MACB 



approach to identify outcomes 
and indicators: Generation of 
Input, Consensus Working 
Session, and Synthesis of 
Consensus. The three stages 
and the tasks that must be 
accomplished during each 
stage are shown in Figure 2. 

Before beginning the process, a 
conceptual model that includes 
a basic framew 7 ork of outcomes 
must be delineated for partici- 
pants to give input on and for 
them to develop and come to 
consensus on a list of indicators 
of the outcomes. The frame- 
work that NCEO used is shown 
in Figure 1 . NCEO's goal was to 
obtain input on the model and 
to have the participants 



NCEO used the MACB process 
to help generate and reach 
agreement on the outcomes and 
indicators that are included in a 
model of educational outcomes 
(Figure I). NCEO produced, 
with input from many individ- 
uals, large lists of outcomes and 
indicators and used MACB to 
determine how important these 
indicators were to various 
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Figure 1 



Conceptual Model of Outcomes 



♦ 



= OUTCOME DOMAIN 



♦ 



Presence and 
Participation 



Resources 
(Input and Context) 



Educational 
Opportunity and 
Process 



Accommodation 
and Adaptation 



♦ 
♦ 
♦ 
♦ 
♦ 
♦ 



Physical Health 



Responsibility and 
Independence 



Contribution and 
Citizenship 



Academic and 
Functional Literacy 



Personal and Social 
Adjustment 



Satisfaction 



develop and come to consensus 
on a list of indicators of the 
outcomes. 

Stage I: Generation of input 

The purpose of the first MACB 
stage is to generate the indica- 
tors. Participants representing 
stakeholder groups need to be 
identified. NCEO recommends 
at least 4 but no more than 10 
individuals be selected, contact- 
ed and asked to participate in a 
MACB working session. 



Depending upon the purpose 
of the meeting, the group can 
be a mixture of several different 
types of stakeholders (e.g., par- 
ents, teachers, administrators) 
or it can be members from just 
one stakeholder group. 

Although groups of more than 
10 were used in some NCEO 
working sessions, discussion 
and group processing were 
quicker and easier to facilitate 
with groups of 10 or less. 



After the participants are iden- 
tified, the next step is to send 
them the current model with 
instructions to review it, write 
approximately four indicators 
for each domain, and return the 
list one week before the MACB 
stage II working session. 

After all the lists have been 
returned, a master list for each 
area must be developed to 
include the respondents' lists, 
as well as indicators developed 
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Figure 2 



Multi-Attribute Consensus Building: 
Stages and Tasks to Accomplish 



Before the Process Begins 

♦ Develop conceptual model 

♦ Determine stakeholder 
groups - These should be 
groups influenced bv the 
finished product. 



tStage I: 

Generation of Input 



1. Select 4 to 10 stakeholders to 
attend the working session. 

2. Have stakeholders generate 
indicators for each outcome 
domain. 

3. Develop a master list of indi- 
cators for each outcome area. 

4. Create a spreadsheet shell 
that computes the average and 
range of the ratings for each 
indicator. 

i 

I -\ Prepare tor stakeholder 
I working session (e.g.. make 
I copies ot ratings sheets, etc.). 



Stage 11: . * ~ 

Consensus Working Session \ 



Introduction 

1 . Explain purpose of meeting. 

2. Describe MACB process. 

3. Tell participants how informa- 
tion will be used. 

Clarification 

1. Present master list of indicators 
for one outcome domain. 

2. Allow participants to make 
modifications to indicators if 
group agrees. Computer operator 
modifies master list. 

3. Limit discussion to wording 
and meaning of indicators. 

Rating 

1. Participants rate the impor- 
tance of each indicator {0 - 1(H)). 

2. One indicator must be 100. 

3. Computer operator enters 
information into spreadsheet. 

Discussion 

1. Review average ratings for 
each indicator with stakeholders. 

2. Discuss indicators with large 
ranges in ratings or mid-range 
average ratings (h0 - 80). 

3. Encourage revisions to ratings 
after discussion. 



tStagelll: 

iSyntheps of Consensus 



1 . Compute standard devia- 
tion, median, and number of 
individuals who rated each 
indicator greater than 75 and 
less than 50. 

2. Identify indicators with very 
high and verv low average rat- 
ings. Keep those with very 
high ratings. 

5. Make decisions about 
remaining unclear items. Use 
other indices and staff judg- 
ment to clarify the importance 
of the indicator. 
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by the staff or previous groups. 
The indicators should all be 
phrased in a similar format, 
eliminating duplicates on the 
master list. See Box A for a 
sample list that NCEO worked 
with during this stage. 

Stage II: Consensus 
working session 

It is in this second MACB stage 
that consensus building occurs. 
At least two staff members 
need to run the meeting, which 
can last from half a day to three 
days. The number of outcomes 
and indicators that need to be 
rated and the number of indi- 
viduals who attend determine 
the length of the meeting. One 
staff member serves as the facil- 
itator/moderator while the 
other operates the computer. 
Both members should be famil- 
iar with the outcomes and 
indicators, and any issues that 
may arise about them. 

To ensure the success of the 
meeting, it is essential to 
explain in detail to participants 
the process and the purpose of 
the meeting. For example, the 
materials in Appendices A and 
B can be revised and given to 
the participants to help intro- 
duce the MACB process. 

It is also important to mention 
how the collected information 
will be used. After this, the 
facilitator can introduce the 
first outcome domain and start 
the consensus-building activity 
with the group. 



Consensus building consists of 
three parts: clarification, 
rating, and discussion. 

During clarification, the facili- 
tator presents the list of indica- 
tors to the group and asks for 
questions about the wording or 
about items individuals do not 
understand. Group members 
may modify the statements if 
the rest of the group agrees. 
The computer operator makes 
changes on the computer that is 
connected to an overhead pro- 
jection display, allowing group 
members to see each change 
that is made. Discussion during 
this time should only be spent 
on clarifying the indicators and 
adding any that are considered 
necessary by the group. 

One caution: The facilitator 
needs to prevent the group 
from focusing too much time 
on one or two words of an indi- 
cator. Clarification is the pur- 
pose here, not wordsmithing. 
And, the focus should be on 
understanding what is meant 
by the items, not on their merit. 
The "understanding" piece is 
important in this step. When 
members do not understand 
specific items, the facilitator can 
call on the person who con- 
tributed the item to explain 
what he or she meant, or ask 
the group to agree on the mean- 
ing of the item. 

Following the clarification step, 
the facilitator begins the rating 
process. Every participant gives 



each of the indicators a rating 
from 0 to 100, where 100 repre- 
sents the most important indi- 
cator. Participants must first 
assign the 100 rating and then 
rate all other indicators relative 
to this indicator. The only 
requirement is that at least one 
indicator must be considered 
the moot important, although 
more than one may be rated 
100. 

Figure 3 is an example of a rat- 
ing sheet with directions. On 
ihis sheet, space is provided for 
the participants to write down 
the items they are rating. Two 
alternatives to writing the items 
are: 

♦ Continue to display the 
indicators, using a second com- 
puter and panel 

♦ Print out lists of the modi- 
fied indicators before beginning 
the rating process. 

As participants complete the 
rating sheet they give it to the 
computer operator, who enters 
the ratings into the spreadsheet 
on the computer, checks to 
make sure that each participant 
has rated at least one indicator 
100, and sees that all partici- 
pants have rated all indicators. 
By doing this, the data en/cry 
process can be finished almost 
immediately after the last sheet 
is received. 

The computer displays the 
spreadsheet with each mem- 
ber's ratings, the group's range, 
and the average for each 



Outcomes & Indicators - NCEO Report 



Consensus Building 



Box A 

List of Indicators Used for 
Consensus-Building Process 

at School Completion Level for Accommodation- Adaptation 



A Percent of individuals who have mastered accommodation-adaptation or 

compensation skills required to move about in their environments (e.g., use a 
wheelchair). 

B Percent of individuals who have mastered accommodation-adaptation or 

compensation skills required to communicate orally or in writing (e.g., use a 
communication board). 

C Percent of individuals who have mastered accommodation-adaptation or 

compensation skills required to read (e.g., Braille, tape recordings). 

D Percent of families who make necessary adjustments to enable their child to access 

educational opportunities (e.g., use community agencies, hire a tutor). 

E Percent of individuals rated as demonstrating behaviors acceptable to their 

environments (e.g.. home, school, and community). 

F. Percent of individuals who appropriately generalize information across settings. 

G Ratings of the level at which family members are able to pursue their own 

activities. 

H Percent of individuals who have mastered skills required to pefform vocational, 

daily living and academic tasks. 

I Percent of families who indicate after the student exits school they are sufficiently 

prepared to cope with student's disability 

/ Percent of individuals whose parents actively participate in the IEP meeting. 

K Percent of students and families who are knowledgeable about resources and 

programs in the community that will be needed after student leaves school. 
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Figure 3 

Example of an MACB Importance Rating Sheet 

Assigning Importance Weights to Indicators of Outcome Domains 

Outcome Domain: 

Your stakeholder group has identified, modified, and confirmed a consolidated set of measurable 
indicators for the outcome domain listed above. Read over the indicators and if necessary write them 
below using their assigned codes (A, E, etc). Once this is accomplished, do the following: 

1) Assign the indicator(s) that you believe is (are) the most important 
a value of 100 points; and then, 

2) Rate each of the other indicators on a scale of importance from 0 
to 100 points. 

Please note: It is permissible to assign more than one indicator a value of 100 points. 



Indicator 



Value 
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indicator. Appendix B includes 
an example of the spreadsheet 
used by NCEO. This format can 
be created with most spread- 
sheet software e>. EXCEL, 
Lotus 123, Quattx-./;. 

After all ratings are entered into 
the computer, the group's 
ratings are displayed on the 
overhead projector. From these, 
the facilitator helps to create 
group discussion about those 
indicators that have a large 
range and a high degree of vari- 
ability in ratings. 

Those who have given low or 
high ratings on a specific indi- 
cator are asked to describe why 
they rated the item as they did. 
This is done especially with 
those individuals who have rat- 
ings that could be considered 
"outliers" (ratings outside the 
typical range). If it appears thai- 
all members are in general 
agreement about a specific indi- 
cator, discussion is not required 
and best to avoid in the interest 
of saving time. 

Throughout the discussion 
process, the facilitator encour- 
ages participants to feel free to 
change their ratings after hear- 
ing other opinions. This step is 
a very important part of the 
process. Therefore, the facilita- 
tor must keep the discussion 
focused on the indicators that 
have discrepancies in ratings. 
After discussing these indica- 
tors, the facilitator moves on to 
the next domain and begins the 




clarification process. This 
three-part procedure continues 
until consensus is reached for 
all domains. 

The dialogue in Box B repre- 
sents what the facilitator might 
do and sav to generate discus- 
sion after ratings have been dis- 
played. Note that the "outliers" 
and other group members have 
the freedom to change or not 
change their ratings. 

Stage III: Synthesis of 
consensus 

After the working session 



comes the final stage of MACB, 
which completes the final selec- 
tion of indicators. The selection 
process is generally based on 
the data, but for borderline 
cases, the decision should be 
based upon statements made 
during the rating process and 
other factors concerning the 
indicator (such as feasibility of 
use or similarity to other indi- 
cators already chosen). 

Several different indices of cen- 
tral tendency and variation can 
easily be produced using 
spreadsheet formulas. Along 



Box B 

Sample Dialogue 

About Indicator with an Average Rating of 75, with 
a Range from 0 to 100 



Facilitator: Indicator C received an average rating of 75. 

But the ratings ranged horn 0, given In/ two 
people, to Wb, given by three people. Both Mary 
ana ]oe rated Cas zero. Mary or foe, or both of 
you, please tell us why you gave Indicator C a 
zero rating. 

[Mary and /or Joe speak] 



Facilitator: Okay, how about you George, or you 

Stephanie, why did you rate C as 100? 



[George and/or Stephanie speak] 



Facilitator: Tlwse are all good points. Does anyone else 

want to wake a statement? Or, does anyone 
want to change a rating after hearing the 
discussion? 
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Box C 



Sample of Decisions Made During Process of Synthesizing Consensus 



icator Avg SD 


Mitt 


iviux 








uectsioti 


A 


65 


30 


0 


100 


65 


50 


50 


Drop 


B 


98 


4 


90 


100 


100 


100 


0 


Keep 


C 


94 


8 


75 


100 


98 


100 


0 


Keep 


D 


79 


22 


50 


100 


90 


57 


29 


Drop 


E 


28 


20 


0 


50 


35 


0 


100 


Drop 


F 


88 


35 


0 


100 


100 


86 


14 


Keep 


G 


73 


30 


10 


100 


83 


63 


25 


Keep 


H 


14 


26 


0 


80 


0 


13 


88 


Drop 


I 


99 


3 


90 


100 


100 


100 


0 


Keep 


J 


75 


37 


10 


100 


85 


63 


25 


Drop 



Note: Indicator J was dropped while Indicator G was kept even though the ratings for both 

indicators were very similar. Staff judgement determined that indicator G was important 
to keep because it was part of an already existing data base that would provide 
information on the outcome. 



with the standard deviation, 
median, range, and mean, 
NCEO suggests using the per- 
centage of individuals who 
rated an indicator higher than 
75 and the percentage who had 
ratings lower than 50 as useful 
indices of a group's consensus 
on a specific indicator. These 
statistics will help decide which 
indicators to keep and which to 
drop. 

Although complex combina- 
tions of statistical criteria could 
be used to direct decision mak- 
ing (Lewis et aL, 1991), NCEO's 
experiences from numerous 
working sessions suggest that a 
simple approach is quite 
adequate for making good 



decisions. Basically, N07.0 
examines the average ratings to 
make the first cut on indicators. 

The general rule is to keep 
items with high average ratings 
(above 90) and drop those with 
low average ratings (below 50). 
When it is not clear whether to 
keep or delete an indicator, the 
decision should generally be 
based on the data, but w 7 hen the 
data are not clear, a judgment 
must be made by the staff. 

The sample of decisions in Box 
C provides an example of the 
decisions made for a set of indi- 
cators that were rated by a 
group and summarized using a 
spreadsheet program. 



Equipment 

One of the unique features of 
MACB is the use of a combina- 
tion of equipment: computer, 
spreadsheet and word process- 
ing software, overhead projec- 
tor, and a computer display 
projection panel to expedite the 
consensus-building process and 
provide feedback to the stake- 
holder group. Different combi- 
nations and configurations are 
possible. See Appendix C for 
descriptions of several IBM and 
Macintosh-based systems. 
When creating a system, 
however, it is important to test 
the equipment before conduct- 
ing a MACB working session 
since what "should work" does 
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not always work. After choos- 
ing a spreadsheet program, also 
listed in Appendix C, gain 
experience and familiarity with 
the selected program to ensure 
successful use. 

Technical and Logistical Issues 

When determining which type 
of computer to use, several fac- 
tors must be considered. 

First, the computer must be 
able to connect to a display 
panel that allows the computer 
screen to be displayed through 
an overhead projector onto a 
projection screen. Some com- 
puters come equipped with 
video output ports, while other 
computers need to have a card 
or cable installed to make them 
ready for use. 

Second, meeting facilitators and 
computer operators need to be 
familiar with the chosen soft- 
ware and the computer. The 
MACB working session com- 
puter operator needs to know 
how to solve problems that 
might arise, and be very adept 
at entering, changing and sav- 
ing text and data. 

The speed of the computer is 
the third factor that must be 
considered. Although a fast 
computer is not essential, the 
process can become painstak- 
ingly slow if it takes an 
extraordinary amount of time 
to enter data and update the 
display. 



Fourth, the physical set-up of 
the room needs to be consid- 
ered when preparing to hold a 
MACB working session. The 
computer equipment might 
interfere with the participants' 
view of the screen(s). Also, 
participants should be seated in 
a way that will promote group 
discussion. Thus, all partici- 
pants should be able to see and 
hear each other. 

Ways to Modify the Process 

Consideration should also be 
given to ways to modify the 
MACB process to best suit the 
situation in which it will be 
used. 

One way to modify the process 
is to increase the number of 
computers and panels. MACB 
works very well with only one 
computer and panel, but by 
using two computers, panels, 
and overhead projectors, the 
indicators and the ratings can 
be displayed simultaneously. 
With two computers, the partic- 
ipants do not need to write 
down the indicators before they 
rate them. Two computers are 
especially useful when the 
amount of time is limited. 

Another way to modify the 
process is to eliminate the com- 
puters. Instead, the indicators 
and ratings can either be writ- 
ten and modified on paper or 
on a chalkboard. Of course, this 
method of computing ranges, 
and group and individual aver- 



age ratings slows down the 
process and makes it more diffi- 
cult to change scores. But if 
computers are not available, 
this process still works effec- 
tively to build consensus. 

The paper and pencil approach 
can also be used in a modified 
MACB process conducted via 
the mail. This method is very 
similar to the Delphi technique 
(Dalkey, 1967), but less time 
consuming and more objective. 
Besides rating items, partici- 
pants write their reasons for 
each rating, and mail them to 
the facilitator who processes all 
the information onto a 
summary form. 

Each participant receives this 
form that contains means, stan- 
dard deviations, ranges for 
items and comments of the 
entire group, as well as the 
individual's original ratings. 
Each participant reviews the 
form and makes changes to the 
material or leaves it unchanged 
before sending it back to the 
organizer. These ratings are 
used to decide which indicators 
to keep and drop. 

Unfortunately, this approach 
loses the possibility of viewing 
the effects of changed ratings 
and eliminates the opportunity 
for further re-rating. But it 
provides an alternative for con- 
sensus building with targo 
groups where it is impossible to 
bring participants together for a 
meeting. 
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Other Uses of the Process 

MACB is well suited for any 
type of group decision making 
that requires developing a list 
and then selecting specific 
items from the list. Although 
NCEO has only used this 
process in selecting educational 
outcomes and indicators, 
MACB has been used by others 
to help define and select 
research priorities, to identify 
educational outcomes of uni- 
versity graduate level training 
programs, and to facilitate 
strategic planning by identify- 
ing critical educational issues in 
the next 20 years. 

Advantages/Limitations 

When compared to other con- 
sensus-building processes, such 
as the Delphi technique, three 
MACB features stand out as 
unique and especially useful. 

First, in the MACB rating pro- 
cedure, everyone hai equal 
input into the final decision and 
an opportunity to detail why 
they rated an indicator a certain 
way. 

Second, the MACB process 
keeps discussion focused on the 
topic at hand. Consequently, 
the important points of the 
issue are more often addressed 
in detail than with other meth- 
ods. Also, by asking the group 
to focus on the ratings, the facil- 
itator has an easier time getting 
the discussion back on track. 



A third advantage of MACB is 
that participants enjoy the 
process and are generally satis- 
fied with the final product. 

MACB ha^- several limitations 
that need to be recognized by 
those planning to use the 
process. As with any type of 
rating system, MACB is suscep- 
tible to different response styles 
of the participants. For exam- 
ple, individuals may use differ- 
ent reference points when giv- 
ing their ratings, always 
respond in a similar fashion, or 
only use a certain part of the 
scale. 

Another limitation is that 
groups often spend a large 
amount of time attempting to 
reword the indicators because 
they are easily modified on the 
computer. Modifying an indica- 
tor is important if the meaning 
becomes clearer to the group. 
However, discussion should 
generally focus on the ratings 
and why one person rated an 
indicator differently from 
another. 

The final limitation of the 
MACB process is that strong 
believers can sometimes domi- 
nate conversation and prevent 
others from participating as 
fully as they desire. One solu 
tion to this problem is to pro- 
vide participants with a form 
for writing down their final 
thoughts, concerns, and reser- 
vations about any indicator, 
and encourage them to use it. 



Conclusion 

For NCEO, the MACB process 
has been invaluable for reach- 
ing consensus about important 
educational outcomes. With it, 
NCEO has brought together 
individuals from very diverse 
perspectives (e.g., legislators, 
school administrators, parents, 
general educators, special edu- 
cators) to reach agreement. 

The MACB process promotes a 
focus at an objective level, even 
when the issues being dis- 
cussed are quite value-laden. 
And, after completing the 
process, the participants sup- 
port the product they have pro- 
duced, and the implementation 
and dissemination of the 
results. ♦ 
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APPENDDCA. ExplanationrofConsensusrBuilding^ Process 



Consensus-Building 
Process for 
Identifying 
Outcome Indicators 

Multi-Attribute Consensus 
Building (MACB) 

The Multi-Attribute Consensus 
Building (MACB) process will 
be used to reach agreement on 
indicators within each outcome 
domain. This process is a modi- 
fication of the Multi-Attribute 
Utility analysis procedure that 
has been used to analyze policy 
v in management sciences, public 
health, and other social services 
needing to structure decision 
making for evaluating program 
alternatives. 

WJiat Are We Going to Do? 

MACB contains three distinct 
stages: 

1) generating indicators 

2) rating indicators 

3) selecting indicators 

When you sent your modifica- 
tion to the current list of indica- 
tors, you had completed the 
first stage. The second stage 
will occur during the meeting 
that you are attending. Our 
goal is to discuss the indicators 
and then to rate them. 



The discussion and rating 
process is fairly simple. We will 
work with only one domain at 
a time, using computers to aid 
the revision and rating process. 

First we will display the list ot 
indicators that we have com- 
piled from our previous list and 
your additions. Then we will 
ask for feedback on the clarity 
of the indicators, modifying the 
list as needed. 

After this brief discussion, you 
will identify the indicator that 
you think is most important 
and give it a rating of 100. Next, 
you will rate the remaining 
indicators in importance from 0 
to 100, relative to the indicator 
given the 100 rating. Additional 
ratings of 100 may be given at 
this time. 



Note: You must give at 
least one indicator a rating 
of 100 in each domain. 



After the indicators have been 
rated and entered into the com- 
puter, they will be displayed for 
the entire group to view (see 
attached sample). As a group, 
we will examine the average 
rating and range of ratings for 
each indicator. 



Discussion will begin with indi- 
cators for which there does not 
seem to be consensus. At the 
end of this discussion, everyone 
will have an opportunity to 
change their ratings based on 
what they hear in the discus- 
sion. This procedure will be 
repeated for each domain. 

The third stage of the consen- 
sus-building process will occur 
after the meeting. We will 
examine the ratings that you 
have produced and use the 
mean rating for each indicator 
and other statistics to help 
decide which indicators are 
retained. 

We will look at the following 
indices when making decisions: 
mean rating, median rating, 
standard deviation of ratings, 
range of ratings, number of 
people giving a rating above 75, 
and number of people giving a 
rating below 50. ♦ 
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Indicators 

Domain: Compensatory and Accommodation Skills - Grade 8 

A. Percent who have mastered skills required to move about in the environment (use of 
wheelchair, cane, etc.). 

B. Percent who have mastered skills required to communicate with others (use of sign 
language, symbol board, etc.). 

C. Number or proportion who comprehend and use age-appropriate communication skills. 

D. Proportion who are able to independently use public transportation (if available in their 
community) for business or personal use (e.g., shopping, recreation). 
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Compensatory Grade 8 
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APPENDIX C; EquipmenfeCotiaFiguiatioi|:i 



Several different types of com- Computer Cable/Connector Panel 

puter, cable, and panel configu- 
rations can be used to project 
computer displays through an 
overhead projector. The num- 
ber of different options and 
types of equipment and soft- 
ware available increases the 
complexity of finding compo- 
nents that are compatible. 
NCEO has used a number of 
different configurations and the 
ones that worked are listed 
here. 

^ftware 

There is a large variety of word 
processing and spreadsheet 
programs that can be used. 
NCEO has used: Microsoft 
Word 5.0 on Macintosh com- 
puters, and Wordperfect 5.1, 
Wordperfect 5.2 for Windows, 
and Microsoft Word 2.0 for 
Windows on IBM compatible 
computers. NCEO has only 
used Excel 4.0 spreadsheet soft- 
ware for Macintosh and IBM 
computers, but Lotus 123, 
Quattro and any other program 
that allows for easy entry, dis- 
play, and calculation of a row of 
numbers will suffice. ♦ 
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Macintosh Plus 
with A10 adapter 
NOTE: Very slow system 

Macintosh Classic 
with A21 adapter 

Macintosh Powerbook 
160 or 180 with 
Video Output port 

IBM or compatible 
386 or 486 Portable 
with Video Output 
Port 
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